Method and system for detecting malicious payloads

ABSTRACT

Disclosed is an improved method, system, and computer program product for identifying malicious payloads. The disclosed approach identifies potentially malicious payload exchanges which may be associated with payload injection or root-kit magic key usage.

BACKGROUND

In recent years, it has become increasingly difficult to detectmalicious activity carried on networks. The sophistication of intrusionshas increased substantially, as entities with greater resources, such asorganized crime and state actors, have directed resources towardsdeveloping new modes of attacking networks.

For example, in enterprise networks it is critical to restrict useraccess to servers located inside data centers. One pernicious type ofintrusion pertains to the situation when an outside entity attempts todeliver a malicious payload onto a host within an internal network.

With conventional solutions, perimeter security tools such as statefulfirewalls are used to define explicit rules about what traffic isallowed to be sent and received from external servers. While theserule-based approaches are able to deter some attackers that sendmalicious payload that match one of the pre-defined rule conditions, theproblem is that these rules can be circumvented merely by using apayload delivery method that does not match any of the existing rules.With the constant evolution of the tools and approaches that are used byattackers, this means that the conventional approaches to implementfirewalls are almost certain to include rules that can be circumventedby new approaches used by at least some attackers.

As is evident, there is a great need for approaches that effectively andefficiently identify malicious payloads being delivered to a host on anetwork.

SUMMARY

Some embodiments provide an improved method, system, and computerprogram product for identifying malicious payloads. The embodiments ofthe invention can identify potentially malicious payload exchanges whichmay be associated with payload injection or rootkit magic key usage. Thedisclosed invention can expose when a network is undergoing a targetednetwork attack.

Other additional objects, features, and advantages of the invention aredescribed in the detailed description, figures, and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates an example environment in which a detection enginemay be implemented to perform detection of malicious payload asaccording to some embodiments.

FIG. 1B shows a more detailed view of the detection engine, at whichnetwork traffic is examined to learn the behavior of various activitieswithin the network and to detect anomalies from the normal behavior.

FIG. 2 shows a flowchart of an approach to implement the learningprocess according to some embodiments of the invention.

FIG. 3 shows a flowchart of an approach to implement the detectionprocess according to some embodiments of the invention.

FIG. 4 shows a flowchart of an approach to implement an updating processaccording to some embodiments of the invention.

FIG. 5 is a block diagram of an illustrative computing system 1400suitable for implementing an embodiment of the present invention forperforming intrusion detection

DETAILED DESCRIPTION

Various embodiments of the methods, systems, and articles of manufacturewill now be described in detail with reference to the drawings, whichare provided as illustrative examples of the invention so as to enablethose skilled in the art to practice the invention. Notably, the figuresand the examples below are not meant to limit the scope of the presentinvention. Where certain elements of the present invention can bepartially or fully implemented using known components (or methods orprocesses), only those portions of such known components (or methods orprocesses) that are necessary for an understanding of the presentinvention will be described, and the detailed descriptions of otherportions of such known components (or methods or processes) will beomitted so as not to obscure the invention. Further, the presentinvention encompasses present and future known equivalents to thecomponents referred to herein by way of illustration.

Various embodiments of the invention are directed to a method, system,and computer program product for detecting malicious payloads usingunsupervised clustering.

FIG. 1A illustrates an example environment in which a detection engine106 may be implemented to perform detection of malicious payloads asaccording to some embodiments. Here, an example network 102 comprisesone or more hosts (e.g. assets, clients, computing entities), such ashost entities 114, 116, and 118, that may communicate with one anotherthrough one or more network devices, such as a network switch 109. Thenetwork 102 may communicate with external networks through one or morenetwork border devices as are known in the art, such as a firewall 103.For instance, host 114 may communicate with an external server on node108 through network protocols such as ICMP, TCP and UDP.

However, it is possible that the external server 108 is an attacker nodethat seeks to send malicious payloads to the host 114. By way ofexample, two methods which an attacker may choose to penetrate inside anetwork through the firewall 103, while undetected, are the directpayload injection approach and the backdoor using kernel rootkitapproach.

In the case of direct payload injection, the attacker node 108 sends auniquely marked payload to targeted machines (such as host 114) andinstructs the machine to execute the given code (e.g., which can open aseparate connection back to a machine controlled by the attacker). Thisresults in full access to the targeted machine allowing a direct pathfor exfiltration or subsequent lateral movement. The detection of suchpayloads is extremely difficult as the payloads can be sent over anyopen port and appear as connections with no response from the targetedmachine.

Another possible approach is for an attacker to create a rootkitback-door for future access to the targeted machine. The back-door willparse all incoming traffic listening for a specific payloadcorresponding to a “magic key.” When the root-kit parses the attacker'spayload, the server gives a compliant response and allows the attackerfull access to the targeted machine. Just as with the code injectionapproach, an attacker in the rootkit back-door approach can send itspayload to any open port. Attackers can use ports with active servicesto disguise their actions. The detection of such payloads is extremelydifficult as the payloads and appear as connections with nearzero-length requests with a benign response or no response from thetargeted machine.

These payloads, if left undetected, allow attackers to persist inside anetwork even when they have been identified and removed from othermachines.

As described in more detail below, the detection engine 106 operates byperforming unsupervised machine learning using data about the networktraffic within the network 102. The detection engine 106 operates toidentify malicious payloads used by attackers by considering theiranomalous behavior relative to normal traffic over a machine's activeports. In this manner, the disclosed invention provides an approach toidentify, in real time, the use of malicious payloads, such as forexample, those used in rootkit or code injection attacks insideenterprise networks.

The advantage of this approach is that it does not require a set ofpre-defined rules to be configured and constantly updated to identifythe presence of malicious payloads. Moreover, since the inventionoperates by learning the behavior of the network, this means that thedetection system is effective even when the attackers are constantlyevolving new attack approaches and techniques.

The detection engine 106 may be configured to monitor or tap the networkswitch 106 to passively analyze the internal network traffic in a waythat does not hamper or slow down network traffic (e.g. by creating acopy of the network traffic for analysis). In some embodiments, thedetection engine 106 is an external module or physical computer that iscoupled to the switch 106. While in some embodiments, the detectionengine 106 may be directly integrated as an executable set ofinstructions into network components, such as the switch 106 or afirewall 103. While still, in some embodiments, the detection engine 106may be integrated into one or more hosts in a distributed fashion (e.g.each host may have its own set copy of the distributed instructions andthe hosts collectively agree to follow or adhere to instructions perprotocol to collect and analyze network traffic). In some embodiments,the detection engine 106 can be implemented within one or more virtualmachine(s) or containers (e.g., operating system levelvirtualized-application containers, such as Docker containers and/or LXCcontainers) sitting on one or more physical hosts. Still in someembodiments, the detection engine 106 may be integrated into a singlehost that performs monitoring actions for the network 102.

In the illustrated environment, the hosts may connect to one anotherusing different network communication protocols such as ICMP, TCP orUDP. The detection engine 106 may be configured to work as a passiveanalysis device that can receive, store, and analyze all network trafficsent/received by a single host, or a multitude of hosts. In someembodiments, all network communications may be passed through the switch106 and the detection engine 106 may tap or span (TAP/SPAN) the switch106 to access and create a copy of the network communications; forexample, by performing a packet capture and saving the networkcommunications in one or more packet capture files. Once received, thenetwork communications may be parsed into flows that correspond tosessions.

FIG. 1B shows a more detailed view of the detection engine, at whichnetwork traffic is examined to learn the behavior of various activitieswithin the network and to detect anomalies from the normal behavior.This provides an effective way of determining whether or notcommunication to a host is hiding a payload injection or rootkit magickey.

In some embodiments, the detection engine performs the following tasksfor each machine in the network:

-   -   Perform metadata extraction of the client-server sessions    -   Build clusters of representative sequences for the payloads sent        from clients and servers following established connection for        every active destination port in the network for a defined        learning phase.    -   Flag client-server sequences which differ significantly from        baselined behavior after the learning phase    -   Continuously identify new normal client-server sequences after        the learning phase

At module 130, data from the network traffic is collected, and metadataprocessing is performed upon that data. In particular, every internalnetwork communication session is processed through a parsing module anda set of metadata is extracted. This collected metadata containsinformation about the source and destination machines, whether theconnection was successfully established, the amount of data transferred,the connection's duration, and the client and server's first payloadsfollowing normal connection establishing exchanges.

Therefore, some examples of data to be collected for the network trafficcould include:

-   -   Information for clients and servers, such as IP address and host        information    -   Payloads for both clients and servers    -   Amount of data being transferred    -   Duration of communications    -   Length of time delay between client request and server response

At module 132, the metadata derived from the network communications isexamined to learn the behavior of the network traffic. In someembodiments, this is accomplished by deriving abstractions of theclient-server handshakes following the establishment of a normalconnections between clients and servers. These abstractions are thenused as vectors for storing baseline behavior models of the clients'activities and the servers' activities. In this way, the learning modulecreates a uniform and protocol-agnostic model for identifying known andfuture malicious payloads. Further details regarding an approach toimplement the learning process is described below in conjunction withthe description of FIG. 2.

At module 134, the system performs detection of suspicious activity withregards to payload deliveries. The metadata for network traffic isanalyzed against the models developed by the learning module to identifyanomalies from the normal, baseline behavior. The detection analysis maybe performed either in real-time or on an asynchronous basis. Furtherdetails regarding an approach to perform detection analysis is describedbelow in conjunction with the description of FIG. 3.

At module 136, the system may update the models to reflect updatedcommunications patterns within the network. Further details regarding anapproach to perform the updating process is described below inconjunction with the description of FIG. 4.

Any of the data used or created within this system, such as thecollected metadata, the baseline behavior models, and/or the analysisresults, may be stored within a computer readable storage medium. Thecomputer readable storage medium includes any combination of hardwareand/or software that allows for ready access to the data that is locatedat the computer readable storage medium. For example, the computerreadable storage medium could be implemented as computer memory and/orhard drive storage operatively managed by an operating system, and/orremote storage in a networked storage device. The computer readablestorage medium could also be implemented as an electronic databasesystem having storage on persistent and/or non-persistent storage.

FIG. 2 shows a flowchart of an approach to implement the learningprocess according to some embodiments of the invention. This process isused to identify the normal client and server payloads andrepresentative sequences for each, on all active destination ports.

An unsupervised clustering algorithm is applied to identifyrepresentative sequences for a given combination of parameters. Forexample, in some embodiments, the learning is performed on the basis ofeach unique combination of a server and port. Therefore, for eachserver/port combination, the learning process will identify the “normal”payload characteristics, e.g., for a client payload and a serverpayload. This provides a baseline model of the payload characteristicsthat can later be used to check whether there are deviations from thatnormal behavior.

While the current example approach is used to process the data on thebasis of the client and server payloads for every unique machine anddestination port in the network, it is noted that other combinations maybe utilized in alternate embodiments of the invention. For example, theinventive concepts are also applicable to combinations such asclient/server/port combinations, client/server combination, client/portcombinations, or just ports by themselves.

For each server/port combination, the process at 202 receives thecommunications to be analyzed. In some embodiments, the analysis isperformed on a session basis, e.g., where the analysis is performed onthe collection of packets for an entire TCP conversation. It is notedthat the inventive concepts may also be applied to other protocols andgranularities of network traffic as well.

An abstraction is generated of the client-server handshakes for thesession. The client-server handshake is represented as a sequence ofbytes, where each sequence corresponds to the beginning bytes of therespective client or server payload. In this way, only the first n bytesof the communications need to be analyzed to classify and analyze thecommunications. Alternatively, the entirety of the communications may beanalyzed.

The normal sequences can then be learned over the course of a specifiednumber of sessions and time duration. During this time, new client orserver sequences are added when they differ from previously identifiedsequences more than a defined similarity score.

At 204, a determination is made whether the sequence under examinationis for the first communications to be analyzed. If so, then in someembodiments, this first sequence is stored as the baseline pattern at210.

The process then returns back to 202 to select the next communicationsto analyze. For the next communications, since it is not the firstcommunications, the process proceeds to 206 to compare against the oneor more baseline patterns.

A similarity score can be calculated to perform the comparison. In someembodiments, the similarity score is calculated using a Hamming distancebetween two sequences with a bias towards sequential equivalences.

If the similarity is not within a threshold distance, then the newsequence is assumed not to cluster with the previously identifiedbaseline pattern(s). Therefore, at 210, the sequence is stored as a newbaseline pattern. In some embodiments, in order to ensure that aconverged normal state is reached, a minimum number of sessions and timemust pass without identifying sequences as new baseline patterns.

If the similarity is within the threshold distance, then the newsequence is assumed to cluster with the previously identified baselinepattern(s). Therefore, no new baseline is added for this situation.Instead, the process will move to 212 to check whether there are morecommunications to process.

If there are further communications to analyze, then the process returnsback to 202 to receive the next communications. If there are no furthercommunications to analyze, then the process ends the learning phase at214.

FIG. 3 shows a flowchart of an approach to implement the detectionprocess according to some embodiments of the invention. This process isused to flag client-server payload representations which differsignificantly from the baselined behavior(s).

At 302, the communications to be analyzed are received. As previouslynoted, the analysis is performed on a session basis. A sequence of bytesfor the payload is extracted for the detection analysis process.

At 304, a comparison is made between the extracted sequence of thecurrent communications and the patterns that have been saved for thebaseline behaviors. In particular, the client and server payloads in allsessions following the learning period are scored relative to thelearned normal behavior. As before, a Hamming distance can be used tocalculate the score.

A determination is made at 306 whether the calculated score is within adefined threshold. If the pair of client-server sequences differ morethan the specified similarity score, then the session containing theexchange is flagged as a potentially malicious payload at 308.

If there are further communications to analyze (310), then the processreturns back to 302 to receive the next communications. If there are nofurther communications to analyze, then the process ends the detectionphase.

FIG. 4 shows a flowchart of an approach to implement an updating processaccording to some embodiments of the invention. This process is used toidentify new normal client and server payloads. It is noted that thisprocess is employed since new client and server payloads can be learnedby the system even after the learning period.

At 402, the communications to analyze are received. As previously noted,the analysis is performed on a session basis for server/portcombinations. A sequence of bytes for the payload is extracted for thedetection analysis process.

At 404, over a given period of time, a determination is made whetherthere is a new sequence having a score that is below the similaritythreshold that is consistent over that period of time.

At 406, if only one of the client or server sequences is scored belowthe similarity threshold but the other is above, the new sequence isadded as normal behavior. If pairs of client-server sequences areflagged more than a specified number of times as malicious, they arelearned as normal behavior for the specific port.

Therefore, what has been described is an improved method, system, andcomputer program product for identifying malicious payloads. Thedisclosed invention can identify potentially malicious payload exchangeswhich may be associated with payload injection or root-kit magic keyusage. The disclosed invention can expose when a network is undergoing atargeted network attack.

System Architecture Overview

FIG. 5 is a block diagram of an illustrative computing system 1400suitable for implementing an embodiment of the present invention forperforming intrusion detection. Computer system 1400 includes a bus 1406or other communication mechanism for communicating information, whichinterconnects subsystems and devices, such as processor 1407, systemmemory 1408 (e.g., RAM), static storage device 1409 (e.g., ROM), diskdrive 1410 (e.g., magnetic or optical), communication interface 1414(e.g., modem or Ethernet card), display 1411 (e.g., CRT or LCD), inputdevice 1412 (e.g., keyboard), and cursor control. A database 1432 may beaccessed in a storage medium using a data interface 1433.

According to one embodiment of the invention, computer system 1400performs specific operations by processor 1407 executing one or moresequences of one or more instructions contained in system memory 1408.Such instructions may be read into system memory 1408 from anothercomputer readable/usable medium, such as static storage device 1409 ordisk drive 1410. In alternative embodiments, hard-wired circuitry may beused in place of or in combination with software instructions toimplement the invention. Thus, embodiments of the invention are notlimited to any specific combination of hardware circuitry and/orsoftware. In one embodiment, the term “logic” shall mean any combinationof software or hardware that is used to implement all or part of theinvention.

The term “computer readable medium” or “computer usable medium” as usedherein refers to any medium that participates in providing instructionsto processor 1407 for execution. Such a medium may take many forms,including but not limited to, non-volatile media and volatile media.Non-volatile media includes, for example, optical or magnetic disks,such as disk drive 1410. Volatile media includes dynamic memory, such assystem memory 1408.

Common forms of computer readable media includes, for example, floppydisk, flexible disk, hard disk, magnetic tape, any other magneticmedium, CD-ROM, any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, RAM, PROM, EPROM,FLASH-EPROM, any other memory chip or cartridge, or any other mediumfrom which a computer can read.

In an embodiment of the invention, execution of the sequences ofinstructions to practice the invention is performed by a single computersystem 1400. According to other embodiments of the invention, two ormore computer systems 1400 coupled by communication link 1415 (e.g.,LAN, PTSN, or wireless network) may perform the sequence of instructionsrequired to practice the invention in coordination with one another.

Computer system 1400 may transmit and receive messages, data, andinstructions, including program, i.e., application code, throughcommunication link 1415 and communication interface 1414. Receivedprogram code may be executed by processor 1407 as it is received, and/orstored in disk drive 1410, or other non-volatile storage for laterexecution.

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the invention. Forexample, the above-described process flows are described with referenceto a particular ordering of process actions. However, the ordering ofmany of the described process actions may be changed without affectingthe scope or operation of the invention. The specification and drawingsare, accordingly, to be regarded in an illustrative rather thanrestrictive sense.

What is claimed is:
 1. A method for detecting a malicious networkpayload, comprising: collecting network traffic corresponding tocommunications within a network; extracting metadata from the networktraffic for a client-server session; in a learning phase, identifyingfrom the metadata any representative sequences for payloads sent betweena client and a server for the client-server session; and in a monitoringphase, detecting for any client-server sequences corresponding to apayload that are different from the representative sequences identifiedin the learning phase.
 2. The method of claim 1, wherein therepresentative sequences are identified on the basis of a uniquecombination of the server and a port, where baseline payloadcharacteristics are identified for the unique combination of the serverand the port.
 3. The method of claim 2, further comprising: receiving afirst communication for the unique combination of the server and theport; setting the baseline payload characteristics in correspondence tothe first communications; for an additional communication, comparing theadditional communication against the baseline payload characteristics toobtain a similarity score, and identifying new baseline payloadcharacteristics that corresponds to the additional communication if thesimilarity score is not within a threshold distance and a minimum numberof sessions or time has elapsed.
 4. The method of claim 1, whereinidentification of the representative sequences in the learning phase isperformed by constructing one or more models corresponding to baselinebehavior for the client and the server.
 5. The method of claim 4,wherein new normal client-server sequences are identified even after aninitial learning phase by updating the one or more model.
 6. The methodof claim 1, wherein the monitoring phase is implemented by: receiving anew communication for analysis; comparing an extracted sequence for thenew communication against the representative sequences identified in thelearning phase to generate a score; determining if the score exceeds athreshold; and identifying the new communication as a potential threatfor malicious payload if the score exceeds the threshold.
 7. The methodof claim 1, wherein the metadata extracted from the network trafficcomprises some or more of IP address information for the client and theserver, payload information, quantity information for transferred data,duration of communications, or length of time delay between a clientrequest and a server response.
 8. A system for detecting a maliciousnetwork payload, comprising: a processor; a memory for holdingprogrammable code; and wherein the programmable code includesinstructions collecting network traffic corresponding to communicationswithin a network; extracting metadata from the network traffic for aclient-server session; in a learning phase, identifying from themetadata any representative sequences for payloads sent between a clientand a server for the client-server session; and in a monitoring phase,detecting for any client-server sequences corresponding to a payloadthat are different from the representative sequences identified in thelearning phase.
 9. The system of claim 7, wherein the representativesequences are identified on the basis of a unique combination of theserver and a port, where baseline payload characteristics are identifiedfor the unique combination of the server and the port.
 10. The system ofclaim 9, wherein the programmable code further includes instructionsfor: receiving a first communication for the unique combination of theserver and the port; setting the baseline payload characteristics incorrespondence to the first communications; for an additionalcommunication, comparing the additional communication against thebaseline payload characteristics to obtain a similarity score, andidentifying new baseline payload characteristics that corresponds to theadditional communication if the similarity score is not within athreshold distance and a minimum number of sessions or time has elapsed.11. The system of claim 7, wherein identification of the representativesequences in the learning phase is performed by constructing one or moremodels corresponding to baseline behavior for the client and the server.12. The system of claim 11, wherein new normal client-server sequencesare identified even after an initial learning phase by updating the oneor more model.
 13. The system of claim 7, wherein the monitoring phaseis implemented by: receiving a new communication for analysis; comparingan extracted sequence for the new communication against therepresentative sequences identified in the learning phase to generate ascore; determining if the score exceeds a threshold; and identifying thenew communication as a potential threat for malicious payload if thescore exceeds the threshold.
 14. The system of claim 7, wherein themetadata extracted from the network traffic comprises some or more of IPaddress information for the client and the server, payload information,quantity information for transferred data, duration of communications,or length of time delay between a client request and a server response.15. A computer program product embodied on a computer readable medium,the computer readable medium having stored thereon a sequence ofinstructions which, when executed by a processor, executes a methoddetecting a malicious network payload, comprising: collecting networktraffic corresponding to communications within a network; extractingmetadata from the network traffic for a client-server session; in alearning phase, identifying from the metadata any representativesequences for payloads sent between a client and a server for theclient-server session; and in a monitoring phase, detecting for anyclient-server sequences corresponding to a payload that are differentfrom the representative sequences identified in the learning phase. 16.The computer program product of claim 15, wherein the representativesequences are identified on the basis of a unique combination of theserver and a port, where baseline payload characteristics are identifiedfor the unique combination of the server and the port.
 17. The computerprogram product of claim 16, wherein the sequence of instructions which,when executed by a processor, further performs: receiving a firstcommunication for the unique combination of the server and the port;setting the baseline payload characteristics in correspondence to thefirst communications; for an additional communication, comparing theadditional communication against the baseline payload characteristics toobtain a similarity score, and identifying new baseline payloadcharacteristics that corresponds to the additional communication if thesimilarity score is not within a threshold distance and a minimum numberof sessions or time has elapsed.
 18. The computer program product ofclaim 15, wherein identification of the representative sequences in thelearning phase is performed by constructing one or more modelscorresponding to baseline behavior for the client and the server. 19.The computer program product of claim 18, wherein new normalclient-server sequences are identified even after an initial learningphase by updating the one or more model.
 20. The computer programproduct of claim 15, wherein the monitoring phase is implemented by:receiving a new communication for analysis; comparing an extractedsequence for the new communication against the representative sequencesidentified in the learning phase to generate a score; determining if thescore exceeds a threshold; and identifying the new communication as apotential threat for malicious payload if the score exceeds thethreshold.
 21. The computer program product of claim 15, wherein themetadata extracted from the network traffic comprises some or more of IPaddress information for the client and the server, payload information,quantity information for transferred data, duration of communications,or length of time delay between a client request and a server response.